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(54) Method of producing a stereoscopic image 

(57) A method of producing a stereo image of a (real 
or simulated) scene using at least one (real or simulat- 
ed) camera, wliich creates the impression of being a 3D 
image when viewed on a display by a user, wherein the 
depth of the scene is mapped onto a maximum per- 
ceived depth of the image on the display, and the max- 
imum perceived depth is chosen to provide comfortable 
viewing for the user. 
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Description 

■ [0001] The invention relates to the production of stereo images. 

[0002] The invention has applications in the fields of natural image generation by film or digital photography, analogue 
5 or digital video, movie film generation, and synthetic image generation using methods including computer graphics or 

image based rendering systems, and is particularly relevant to two view stereoscopic devices Including electronic and 

hard copy devices where more than one Image of a scene is generated to create a 3D effect by showing a different 
—image to a viewer's left and right eyes. 

[0003] Applications of the invention include photography, videography, movie production, electronic shopping kiosks, 
10 computer games systems for home or public use, multimedia packages eg encyclopaedias, medical imaging, CAD/ 

CAM systems, scientific visualisation, remote manipulation, remote sensing, security systems and any other application 

where a benefit is found from a stereoscopic 3D image of a scene. 

[0004] IVIany types of stereoscopic and auto-stereoscopic electronic displays and printing or photographic reproduc- 
tion methods have been developed, for example see the following European and British patent applications; EP 0 602 

15 934, EP 0 656 555, EP 0 708 351 , EP 0 726 483, GB 961 9097.0 and GB 9702259.4. The problem of image generation 
for these systems Is less well understood and many existing stereoscopic images can be uncomfortable to view even 
on a high quality stereoscopic imaging device. (Where the term stereoscopic is used it should also be taken to imply 
multi-view systems where more than one image is generated and presented to the user even if only two of the images 
are viewed at any one time by the left and right eyes.) 

20 [0005] As described in B. E. Coutant and G. Westheimer, "Population distribution of stereoscopic ability", Opthal. 
Physiol. Opt., 1993, Vol 13, January, up to 96% of the population can perceive a stereoscopic effect and up to 87% 
should easily be able to experience the effect on desktop 3D display systems. The following summarises some problems 
inherent in previous approaches to stereoscopic image generation. 

[0006] Stereoscopic systems represent the third dimension, depth in front of and behind the image plane, by using 
25 image disparity as illustrated in Figure 1 . The image disparity displayed on a screen has a physical magnitude which 

will be termed screen disparity. Crossed disparity, dfg, results in a perceived depth, N, in front of the display plane while 

uncrossed disparity, dF, results in a perceived depth, R behind the display plane as illustrated in Figure 1. 

[0007] The screen disparities dn or df between homologous points in the left and right images are seen by the viewer 

as perceived depths N or F in front or behind the display plane. To see this effect the viewer must maintain focus on 
30 the display plane while verging their eyes off the display plane. This is thought to stress the visual image if the perceived 

depth value is too great and therefore limits are required for the values of N and F if comfortable images are to be 

produced. 

[0008] These type of stereoscopic display systems do not exactly match the user's perception in the real world In 
that it requires the user to accommodate (focus) on the display surface while verging their eyes away from the display 

35 surface, see Figure 1 , Since the accommodation and vergence mechanisms are linked in the brain (see D. B. Diner 
and D. H. Fender, "Human engineering in stereoscopic viewing devices", 1993, Plenum Press, New York, ISBN 
0-306-44667-7, and M. Mon-Williams, J. P. Wann, S. Rushton, "Design factors in virtual reality displays", Journal of 
SID, 3/4, 1995) this requires some effort from the viewer and a greater effort the more depth is being perceived. The 
Invention recognises that the key variable to control is perceived depth, the larger this value is the more stress is placed 

40 on the viewer's visual system. 

[0009] It is now widely recommended (see L. Hodges, D, McAllister, "Computing Stereoscopic Views", pp71-88, in 
Stereo Computer Graphics and Other True 3D Technologies, D. McAllster, Princeton University Press, 1993; A. R. 
Rao, A. Jaimes, "Digital stereoscopic imaging", SPIE Vol 3639, pp144-154, 1999; and R. Akka, "Converting existing 
applications to support high quality stereoscopy", SPIE Vol 3639, pp290-299, 1999) that images are captured using 

45 two cameras positioned so that the only difference between the two images Is the image disparity due to a horizontal 
translation of cameras. This arrangement is normally referred to as a parallel camera system. This avoids viewer 
discomfort due to keystone distortion (this arises when the cameras are not parallel because the vertical dimensions 
of the two images vary from one side of each image to the other) and associated vertical disparity ie. when the two 
images are superimposed there is varying vertical disparity across the images. In addition, for physical cameras the 

50 optics and light sensitive media must be matched to avoid unnatural intensity or geometric distortions. The latter two 
issues are part of a specific camera design and are not considered further here. 

[0010] As illustrated in Figure 2, the parallel camera image must be processed to ensure the depth range captured 
in the image disparity fits both in front and behind the display plane. This requires the use of offset sensors or film 
behind the lens, skewed camera frustum (in the case of computer graphics) or image cropping. Without such adjust- 
55 ments all depth in the images will be perceived in front of the display plane. 

[0011] Figure 2 shows that the images from parallel cameras need to be adjusted, either to have the edge of the 
image cropped or by the use of an asymmetric camera fustrum. The latter is possible in many computer graphics 
systems, or can be achieved by offsetting the Image sensitive material (eg. CCD or film) to one side in a physical camera. 
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[0012] The factors which directly affect perceived depth are: 
[0013] For depth behind the display surface, uncrossed disparity: 
F = Z/((E/df)-1) 

[0014] For depth in front of the display surface, crossed disparity: 
s N = Z/((E/dn)-t-l) 

[0015] From these equations it can be seen that perceived depth depends on the screen disparity, the viewer's eye 
separation, E, and the display viewing distance, Z. While other methods have approximated or ignored these variables 
the new method allows them to be fully accounted for. 

[0016] The screen disparity (dn or df) is important as it is determined by image disparity which in turn is determined 
10 by the image capture environment, including the camera parameters and the depth in the scene, The invention seeks 
to control image disparity and therefore screen disparity and perceived depth by controlling the camera parameters. 
While various previous methods to control the camera parameters have been proposed none consider the issue of 
directly controlling perceived depth and often approximate the parameters such as the comfortable near and far per- 
ceived depth limits for the target display. 
IS [0017] In S. Kitrosser, "Photography in the service of Stereoscopy", Journal of imaging science and technology, 42 
(4), 295-300, 1 998, a slide rule type calculator is described allowing selection of camera separation given the camera 
details and scene near and far distances. It does not take into account different viewer's eyespacingsand the maximum 
image disparity is set at a predetermined value. It cannot account for the perceived depth the user sees when using a 
particular display and as has been discussed earlier this is the key variable in assessing the comfort of a stereoscopic 
20 image. 

[0018] In L. Lipton, "Foundations of the Stereoscopic Cinema, A Study in Depth", Van Nostran Reinhold Company, 
1982, Lipton examines the mathematics involved in positioning cameras and develops a set of tables for different film 
formats giving maximum and minimum object distances for a given convergence distance and lens focal length. He 
assumes converging cameras will be used and the maximum screen disparity is the same for objects in front of the 

25 screen plane as well as objects behind the screen plane. Two sets of maximum object distances are calculated; the 
first is where a small divergence of the viewer's eyes is allowed (typically 1 degree) and the second where no divergence 
is allowed. These restrictions prevent this method from guaranteeing comfortable image generation at all times. The 
assumption of converging cameras ensures that some vertical disparity will be present in the image and therefore that 
many viewers will find the resulting images uncomfortable to view. 

30 [0019] In the internet site http://www.elsa.com/europe/'press/releases/1 999/graphics/ revelato.htm, Elsa introduce a 
system called 'Dyna-Z' which dynamically adjusts the 'spatial effect. No details are currently available about the method 
of operation of this system, although it is limited to real time computer graphics. 

[0020] The question of what the near and far perceived depth limits should be is partially addressed by existing 
human factors work and It Is possible to deduce typical working values for the SLE VPl displays of far limit +60mm, 

35 near limit -50mm from the following studies. 

[0021] In A. Woods, T. Dooherty, R. Koch, "Image Disioriions in Stereoscopic Video Systems", SPIE Stereoscopic 
Displays and Applications IV, 1 993, 36-48, Woods discusses sources of distortion in stereo camera arrangements as 
well as the human factors considerations required when creating stereo images. These experiments show that there 
is a limit in the screen disparity which it is comfortable to show on stereoscopic displays. A limit of 1 0mm screen disparity 

40 on a 1 6" display at a viewing distance of 800mm was found to be the maximum that all 1 0 subjects of the experiment 
could view. 

[0022] In Y. Yeh, L. D. Silverstern, "Limits of Fusion and Depth Judgement in Stersoscopic Color Displays", Human 
Factors, 32(1), 1 990, 45-60, Yeh shows the results of experiments carried out in order to determine binocular fusion 
limits. It is found that for comfortable viewing over short periods a maximum disparity of 27 minutes of arc is acceptable. 
45 With longer periods of viewing it is possible to adapt to view greater disparities but this does not indicate that large 
disparities are suitable for long term comfortable viewing of stereoscopic displays. 

[0023] In S. Pastoor, "Human Factors In 3D Imaging", experiments by Pastoor indicate that disparities up to 35 
minutes of arc do not cause any discomfort. 

[0024] The invention provides a method of producing a stereo image of a (real or simulated) scene using at least 
50 one (real or simulated) camera, which creates the impression of being a 3D image when viewed on a display by a user, 
wherein in each position at which an image is captured the camera axis is parallel to the camera axis at all other 
positions, and the depth of the scene is mapped onto a maximum perceived depth of the image on the display, and 
the maximum perceived depth is chosen to provide comfortable viewing for the user 

[0025] The method may make use of parameters of the display, including the maximum perceived depth of an object 
55 in front of the display N, and the maximum perceived depth of an object behind the display R 

[0026] The method may make use of the user's eye separation E, and the distance Z of the viewer from the display 
[0027] In an embodiment of the invention, the distance Z' from the camera to the Zero Disparity Plane is calculated 
based on the values of N and F, and also on the values of the distance N' from the camera to the closest surface in 
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the scene, and the distance F' from the camera to the furthest surface in the scene. 

[0028] In a further embodiment of the Invention, the distance Z' from the camera to the Zero Disparity Plane is 
specified, and the values of N' and F' (as defined above) are also specified, and wherein the most suitable of N' or F' 
is kept fixed and a new value is calculated for the other of N' or F' based on the values of N and F 

5 [0029] Any one of N', F and Z' may be calculated based on the values of the other two, 

[0030] In a further embodiment of the invention, the maximum crossed and uncrossed disparities dj^ and dp, are 
calculated from N, F, the separation E of the user's eyes, and the distance Z from the user to the display. 
[0031] In a further embodiment of the invention, the camera separation A used to produce the stereo image is cal- 
culated based on N and F to achieve the desired perceived depth (N + F). 

10 [0032] In a further embodiment of the invention, the field of view of the camera is adjusted to allow for that part of 
the Image which will be cropped as a result of the use of parallel camera positions. 

[0033] In a further embodiment of the invention, the camera separation is fixed, and the desired focal length of the 
camera is calculated to achieve the desired perceived depth (N + F). 

[0034] In a further embodiment of the invention, the focal length f of the camera is taken into account when measuring 
15 the values of N' and F' (as defined above). 

[0035] In a further embodiment of the invention, the camera separation A is limited to the user's eye separation E 
multiplied by the scale factor S as herein defined. 

[0036] The invention also provides a computing device adapted to carry out a method as described above. 
[0037] The invention also provides a camera or camera system comprising such a computing device. 

20 [0038] in order to produce stereoscopic 3D images which are comfortable to view the positioning of cameras (real 
or synthetic) requires great care. Stereoscopic displays are only capable of displaying a limited perceived depth range 
while there may be almost any depth range in a scene to be captured. The new method allows accurate positioning 
of parallel cameras for capturing stereoscopic images which exactly fit the depth in the scene into the perceived depth 
limits of the display. Information about the stereoscopic display for which the images are intended is used, along with 

25 information about the viewer of the display, the scene to be captured and the camera type to be used. 

[0039] A chosen depth range in a scene being imaged is exactly mapped to a pre-defined perceived depth range on 
a stereoscopic 3D display. This ensures that a comfortable perceived depth range such as defined by human factors 
considerations is never exceeded when viewing the stereoscopic 3D images. 

[0040] The near and far limits of the perceived depth range can be set independently allowing for precise control of 
30 the amount of depth effect seen by the user. This allows the images to be adapted to variations in viewer's perception 
of crossed and uncrossed screen disparities and enables precise artistic control of the 3D image composition. 
[0041] In companson to approximate methods or trial and error the new method of camera control ensures that a 
scene is always mapped onto the defined perceived depth range. This results in an easy to use method which does 
not require numerous image adjustments to ensure the depth viewed is within comfortable limits. 
35 [0042] The new method can be implemented In software or hardware allowing it to control physical digital or photo- 
graphic cameras or synthetic computer graphic cameras. 

[0043] Depth measurements are taken to the known film plane (In the case of physical cameras) avoiding problems 
estimating the position of lens aperture positions. This is particularly important with zoom lenses where the apertures 
moves unseen to the user. 

40 [0044] Camera parameters can be calculated instantly, which is useful for real-time computer graphics and control 
of video cameras in particular. Especially when compared with the use by previous methods of slide rules or printed 
tables of results, 

[0045] Parallel cameras are supported which means simple mathematical equations can be used reducing compu- 
tation requirements and the number of control variables. 
45 [0046] Embodiments of the invention will now be described, by way of example only, with reference to the accom- 
panying drawings, in which: 

Figure 1 shows the variables Involved in depth perception in a stereo image; 

50 Figure 2 shows the adjustments which are necessary when using parallel cameras, which are for example image 
cropping or offsetting the image sensitive material (eg CCD or film) to one side in a physical camera; 

Figure 3 shows the viewing geometry of a stereoscopic display; 

55 Figure 4 shows the geometry involved in the capturing of the images of a scene; 

Figure 5 is a flow diagram showing the calculation of camera separation where field of view/focal length is fixed; 
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Figure 6 is a flow diagram showing ttie calculation of camera separation where field of view/focal length is not fixed; 

Figure 7 is a flow diagram showing the calculation of field of view/focal length where the camera separation is fixed. 

[0047] There is only a limited depth range that may be shown on stereoscopic displays, while the depth visible in 
any particular scene could cover almost any range. Therefore, there is a need for a method of controlling the way 
physical scene depth maps to perceived display depth when, for example, talking a stereoscopic photograph or creating 
(ie. computer simulating) a stereoscopic computer graphic image. 

[0048] A first embodiment will be described in which the camera separation, A, is calculated from other input data, 
for stereoscopic photographs tal<en with a physical camera or cameras. 

[0049] We consider the situation where a photographer has composed the desired view, including the camera position 
and focal length (zoom), and wishes to know how far apart to take the left and right image to capture the correct disparity 
range for the target display. This naturally fits into a typical photography session where the photographer composes a 
shot by setting zoom and position before taking the shot. 

[0050] Initially details of the display and viewer are required. These parameters (see Figure 3) include; 
Display parameters 

N maximum perceived depth of any object in front of the display plane 
F maximum perceived depth of any object behind of the display plane 

W Physical width of the viewing area of the display, or partial area In which the image will be displayed. 
Z Viewing distance of the display. 

Viewer parameters 

E Eye spacing of the viewer. 

[0051] Typically these are fixed for each display type. The camera specification may be tailored to a specific viewer 

with an appropriate setting of E - or for an average viewer this can be set to approximately 60mm. 

[0052] This allows calculation of the maximum crossed and uncrossed screen disparities allowable, from N and F: 

ci^ = NE/ (Z-N) Crossed disparity 

cIf=fe/(Z+F) Uncrossed disparity 
[0053] These equations follow from the geometry shown in Figure 3. The ratio df/dp w\\\ be used at a later stage. 
[0054] Following this the known parameters of the camera and scene (see Figure 4) must be entered. These are: 

Scene parameters 

N' Distance from the camera to the closest visible surface in the scene 
F' Distance from the camera to the furthest visible surface in the scene 

Camera parameters 

Wf Width of the imaging area of the camera. 

/ Focal length of the camera/lens required to capture a chosen view. 

[0055] From this information Z', the distance to the Zero Disparity Plane (ZDP) (also known as a "virtual display", 
having width W', see Figure 6) may be calculated. This is the distance from the camera to objects which will appear 
to be in the plane of the display once the photograph has been taken. Anything closer to the camera than Z' will appear 
to be between the display and the viewer. Anything further than Z' from the camera will appear behind the display 
[0056] Since the depth in the scene is to be mapped to the depth allowable on the display the ratio dN'/dp' is equal 
to dN/dp (which we will call R). d^' and dp' are given by the following equations: 
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where A is unknown, so: 



_ (Z'-A/QP 
(F-ZW 



[0057] Which gives, when re-arranged; 



in the case where = dp this simplifies to Z' = 2/(1 /N' + 1/F') 

[0058] With a physical camera it nfiay be difficult to alter the focal length precisely, so the camera separation will be 
calculated assuming / may not be changed. The resulting images will be cropped, giving a stereoscopic image covering 
a slightly smaller field of view than the actual setting input by the photographer. The amount to be cropped is equal to 
the camera spacing /I when the display is mapped to the world. 

[0059] To map the display to the world a scale factor, S, is calculated. The distance to the zero disparity plane Z' is 
known and 6, the field of view, easily calculated, giving the width of the area of the scene captured at a distance Z' 
from the camera. The width of m captured scene at the calculated depth Z' is equal to the effective width W' of the 
virtual display (see Figure 6) plus the camera separation. 

9 =2tan'^ (W/2/) 

(W'+A)= 2 Z'tan(Q/2)=Z'W// [7] 

[0060] The scale factor, S, between the display when mapped to world space and the physical display itself may 
now be calculated: 

S= (W'+A)/W+A/S) = Z'Wy/(W+A/S) [8] 

[0061] Also from equation [3]: 

d,' = Sd, = ^^ [9] 



s z-w 



[0062] Substituting A/S into equation [8] gives: 

S=Z'W^//(W+dNN'/(Z'-N')) 
[0063] And so the desired camera spacing A is given by; 
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[0064] The value of A calculated from cIn' or dp' can be shown to be identical. 

[0065] The correct stereoscopic image is given when a proportion of the left and right images is cropped. The same 
proportion is removed from the left edge of the left image and the right edge of the right image. The proportion to crop 
is given by: 

crop proportion = A/(W' + A) [13] 
[0066] The steps in this method are summarised in the flow diagram of Figure 5. 

[0067] All the input values are independent and may be measured in any order at any time prior to computing the 
stereoscopic camera parameters. As noted later careful calibration of a real camera may be required in orderto ensure 
accurate results. 

[0068] This method has been successfully used as part of an apparatus for taking stereoscopic photographs using 
a single digital still camera, for example a Kodak DCS 3, on a slide rail. The camera is calibrated so that its optical axis 
is at right angles to the slide rail. The view is composed and the focal length read from the camera electronically or 
physically. The inter-axial separation required to take a comfortable Image for the target display is calculated using, 
for example, a camera calculator and the left and right images are captured using this value by sliding the camera 
along the rail the required distance. This apparatus has been successfully used to capture comfortable images for 
various target displays. 

[0069] The calculations specified above assume the distance N' and F' are measured from the pin-hole point of the 
camera. With a real camera it is easier to measure to the film plane so / should be subtracted from N' and F' before 
performing the calculations. 

[0070] A variation of the first embodiment is possible in which the camera separation is calculated from other input 
data, for stereoscopic images created in Computer Graphics software. 

[0071] In this situation all measured distances will be to the appropriate point, so adjusting by /, described in the 
previous section, is unnecessary. In fact field of view is more likely to be defined than focal length so 6 may be entered 
directly into equation [7] above. 

[0072] Further variations of the first embodiment are possible, in which Z' is already chosen, as will now be described. 
1 . If the zero disparity planeZ' and the minimum distance N' are specified, then the resulting F' can be calculated as 



A warning is automatically given to the user by the device performing the calculation that this F' is the maximum 
distance allowable for a visible surface from the camera. 

2. If the zero disparity plane Z' and the maximum distance F' are specified, the resulting N' can be calculated as; 



A warning is automatically given to the user by the device performing the calculation that this N' is the minimum 
distance allowable for visible surfaces to be from the camera. 

3. If the zero disparity plane Z', the minimum distance N' and the maximum distance F' are specified further action 
must be taken since it is not possible to map any arbitrary combination of N', Z' and F' to the allowed perceived 
depth range (ie. N+F) of the display. 
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[0073] Assuming the input value for Z' Is not to be changed the system must choose the most suitable of either N' 
or F' to keep fixed and then compute an adjusted version of the other. 

[0074] The system computes alternative values for N' and F' (here called N* and F*) from equations [14] and [15] 
and compares them with the originals. Either N', Z' and F* will span a range greater than N', Z' and F', or N*, Z' and F' 
5 Will span a greater range. 

[0075] If F*>F' then Z' and N' are used in the camera parameter calculations and F* replaces F' If N*<N' then Z' and 
F' are used in the camera parameter calculations and N* replaces N'. 

[0076] This then ensures that the scene depth range maps onto the comfortable perceived depth range for the display. 
[0077] A second embodiment will now be described In which camera separation is calculated from other Input data, 
10 for stereoscopic photographs taken with a real camera (when 9 , or/, may be altered). 

[0078] In a situation where the field of view of the camera may be altered after setting up the view, with a purpose 
built stereoscopic camera, with an accurately controiled camera, or with Computer Graphics cameras, a view closer 
to the original choice of the photographer may be obtained. 

[0079] In this case equations [1] and [6] are followed as before, in this case the field of view will be expanded to 
15 encompass the extra image width which will later be cropped. 

W = 2Z'tan(Q/2) = Z'W/f [1 6] 

2" [0080] W (the real display width) is known, as is W - this giving the scaling between the display and the world; 

S = W'/W [17] 

^ [0081] The maximum disparities, dfj and cfp are then scaled up to match the size of the virtual display, giving the 
values for df^ and dp. From these A, the camera separation, may be calculated: 

. ^ [18] 

30 

A=^ [19] 

33 [0082] The final stage is to calculate the new field of view for image capture. An area the width of the virtual display 
plus the camera separation must be captured in order to obtain stereoscopic Images with the disparities calculated 
from cameras with parallel axes. The new field of view, e' is: 

40 e' = 2 tan"^ (( IV + -A)/(2'Z')) [20] 

from this a new value for the focal length is found: 
45 r = W/2tan(eV2) [21] 

[0083] All the required camera parameters are now available, specifically 6' and A. 

[0084] Once again, the distances AT and P are assumed to be measured from the pin-hoie point of the camera. If 
they are measured from the film plane then / must be subtracted before carrying out the calculation of A and 6. Sub- 
^° sequently a recursive calculation is required since the focal length is changed by this caicuiatlon. in the recursive 
calculation a new N' and F' are calculated, by subtracting /' from the input values of N' and F'. These new values are 
entered into equation [6] and all subsequent stages, from equation [16] are repeated until convergence. It may be 
possible to solve the recursive equation to give a single calculation producing the correct result without recursion. 
[0085] The steps in this method are summarised in the flow diagram of Figure 6. 

[0086] A variation of the second embodiment is possible in which the camera separation is calculated from other 

input data, for stereoscopic images created in Computer Graphics software (when e (or/) may be altered). 

[0087] With computer graphics cameras everything is usually centred around the pinhole point, so corrections for 
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changes in the focal length are usually unnecessary and the recursive calculation Is unnecessary. 
[0088] 9 will normally be supplied, Instead of /, so is unnecessary and only a new value of 9' needs to be calcu- 
lated, the subsequent calculation of /' is no longer required. 

[0089] The computer graphics methods have been implemented in the SmartStereo sottware package and used to 
5 produce still images, movie sequences and real time computer graphics. These have resulted in easy to use stereo- 
scopic Image generation which repeatably produces comfortable to view stereoscopic images. For example generating 
camera control scripts for the 3D Studio Max software package from Kinetix to produce both still images and movie 
sequences to be replayed in QuickTime movie format. 

[0090] Variations of the first and second embodiments are possible in which camera separation is limited in order to 
■ 10 avoid expanding depth when only a narrow depth range is present in the scene. 

[0091] If a very small depth range is to be captured in a scene, less than the range of allowable perceived depth on 
the display, the objects will appear.to be stretched when the camera separation is adjusted to ensure the scene depth 
maps to the display depth. 

[0092] This can be avoided by clamping the camera separation (A) to a maximum value of the eye separation (E) 
15 multiplied by the scale factor (S). 

[0093] Or setting the scene depth range to capture a volume around the object rather than the object itself, it may 
be desirable to make this volume orthoscopic. 

[0094] In the case of both computer graphics synthetic cameras and physical cameras the perceived depth may 
match exactly the scene depth. In this orthoscopic case there will be minimal depth distortions in the perceived depth 
20 compared to the case where very large or very small scene depth Is mapped to the comfortable perceived depth range, 
[0095] A third embodiment will be described in which the field-of-view of focal length (zoom) is computed from other 
input. 

[0096] In cases where the camera separation Is fixed, the zoom can be adjusted to achieve the desired comfortable 
perceived depth on the display. 

25 [0097] There are advantages to a design of stereoscopic camera in which zoom is controlled by altering the camera 
separation, perhaps when there are two, or more, lenses on the camera to capture the stereoscopic images simulta- 
neously. Other examples are where there is a fixed separation (eg. for AGFA camera + Pentax image splitter) but the 
zoom can be adjusted over a range to help match differing scene depth ranges. This also occurs with certain video 
camera adapters where the zoom may be easier to control than the separation. It also can have a benefit where the 

30 zoom is controlled accurately but the separation is not controlled accurately. The separation is set and measured and 
the zoom adjusted to precisely match the scene depth range to the comfortable perceived depth range on the target 
display. 

[0098] When there is a specified depth to be captured and mapped to a stereoscopic display, and the camera sep- 
aration (A) is specified, the field of view captured (or the focal length of the lens) must be specified to ensure the depth 
35 range displayed on the stereoscopic display within the chosen limits. 

[0099] To calculate appropriate stereoscopic camera parameters in this situation the following method is used: 
[0100] As described in the embodiment above, where camera separation is calculated, compute the value of Z' 
(equations [1] to [6]), the distance to the ZDP, if it is not supplied. 

[0101] In this case A, the camera separation, is known so a value for dN' or dp' is computed (equations [3] and [4]), 
'io The scale factor S is given by: 



S = d,^VdM [22] 
[0102] Scalethedisplaysize.W, to the world, W = SW, and calculate the field-of-view from: 



e' = 2tan"^ ((W +A)/2Z') [23] 

[0103] In this situation measurements for N' and F' are assumed to be to the pin-hole point of the camera. These 
distances are difficult to measure, so a more realistic measurement is to the film plane of the camera. In order to take 
into account the focal length of the lens a recursive calculation would be carried out whereby the focal length is sub- 
tracted from the measured distances (giving new values for N' and F') which are inserted Into equation [6]. A value for 
6' is then calculated (and converted to focal length (zoom)), from equations [22] onwards. This new focal length is used 
to correct the measurements recursively until the result converges. A suitable initial value for the focal length would 
be the focal length needed to give the field of the display as seen by the viewer. It may be possible to solve the recursive 
equation to give a single calculation producing the correct result without recursion. 
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[0104] The steps in this method are summarised in the flow diagram of Figure 7. 

[0105] In a very simple system using Computer Graphics cameras It may be enough just to specify the separation 
and trust the user to offset the cameras from the original composed view in the appropriate way 
[0106]. An improved, easier to use, solution is for the user to provide the position and direction Information for CG 
cameras. The system calculating the camera parameters can then compute the correct position, orientation and fleld- 
of-vlew for the left and right cameras. This is then entered by the user Into the CG software, written to a scrlpi to be 
read directly by the CG software or returned by a programming API directly to the graphics software. 
[0107] Some Computer Graphics systems (APi/software/script languages etc) allow specification of a 'viewing frus- 
tum', This specifies the offset from the viewer to the edges of the image (in world space), which allows Incorporation 
of the cropping, described above, directly into the image generation. When the viewer is positioned centrally in front 
of the display the offset to the left hand edge of the image for the left eye, In world space is: 

fivsti_ = -(W72 - A/2) [24] 
the offset to the right hand edge or the image for the left eye Is: 

frustf, = (W'/2 + A/2) [25] 
the offsets to the top and bottom are (H is the display height scaled to the world): 

frustj = H'/2 [26] 

frustg =HV2 [27] 

[0108] The equivalent values for the right eye are: 

frust^ = (WV2 + A/2) and frusf„ = (W'/2-A/2) [28] 

frustj and frusfg are the same as equations [26] and [27]. 
[0109] These values are normally specified In terms of the near clipping plane, c^, (for example in OpenGL). To 
convert this representation the values above are multiplied by c^/Z' 

[01 10] This type of control has been implemented in SmartStereo OpenGL API which Is used within a VRML model 
viewer to ensure the scene remains comfortable to view at all times. 

[0111] Camera parameters when the viewer Is not positioned centrally In front of the display will now be discussed. 
[0112] If no method of tracking the position of the viewer Is used an assumption must be made about their position 
relative to the display. However, if the viewer's position Is known It may be used to alter the camera parameters to give 
a less distorted stereoscopic Image when not positioned centrally In front of the display 

[01 13] If the viewer's location relative to the centre of the display is specified as follows: (X, Y, Z), where X and Y are 
parallel to the display plane and Z is perpendicularly away from the display, IZl Is used In the camera parameter cal- 
culations, 

[0114] The X and Y position of the viewer relative to the display must be transferred to the world co-ordinate system, 
either by scaling by Sfrom the camera parameter calculation or by scaling by A/E also from the camera parameter 
calculation {x=XSorx = XA/E, and y = YS or y= WE), 

[0115] In the case of Computer Graphics cameras, specified with a frustum (equations [24] to [28]) to the left eye 
frustum is calculated as follows: 
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frustL-=-(WY2-A/2 + x) 

frusta ^ (W'/2 + A/2-x) [29] 
frustT= (H'/2-y) 
fr-ustB = - (H'/2 + y) 

and the right eye frustum is calcuiated as follows: 

frust,. = - (W72 + A/2 + x) 

frustK= (W72-A/2-X) [30] 
frustT= (H'/2-y) 
fnist„ = -(H'/2 +y) 

[0116] If the frustum is specified in terms of the near clipping plane these values are scaled by a^Z 

[01 17] This additional calculation is useful for display systems which tracl< the user's head position and are driven 

by computer graphics systems that can update the view as the viewer moves their head. 

[01 18] Calculations when the input values are maximum and minimum disparities between two existing photographs 

will now be discussed. 

[01 19] This Is the case where a stereoscopic pair (of Images) has been analysed by hardware or software methods 
to find the disparities between homologous points In the images. In this situation some method is required for indicating 
the change in camera separation and the change in the Image cropping necessary to bring the disparities into the 
range defined for the display in use. 

[0120] The details of the intended display device are known, as before, so the disparity range permissible on the 
display may be calculated (c/p- cfy^). These are also converted Into a proportion of the display width by dividing by the 
width. 

[0121] The maximum and minimum disparities present in the images, dp and dfj, are obtained and used to give the 
range present In the stereoscopic images. The disparity range here Is (dp - d/^). These values may be provided In 
many forms, the most Nicely being pixel disparliies, or as a proportion of the image width. When pixel disparities are 
provided the image width in pixels must also be supplied for the following calculations. 

[0122] cfft/i dp, and dp are assumed to be converted to proportions of the display and image widths from here. 
Also the images from which the disparities were tal<en are assumed to have been taken with parallel cameras and to 
have not been cropped. 

[0123] The disparity of points in the input image which should lie on the screen plane, and therefore have zero 
disparity, Is either pre-determined from measurements from the images or calculated in the following manner: 

dz' = (d^' f (d^/dp) * dp') [31] 

[0124] Alternatively, if dz' is specified, one of either d^,' or dp', the missing parameter may be calculated from one of 
the following equations: 



d^' = (dn/dp) (dz' - dp') + dz' 



dF' = d2'-(dp/dN)(dN'-dz') 



[0125] If all three parameters are specified dz' Is retained and d^' or dp' recalculated for the viewing parameters in 
order to cover at least the same depth range, In a similar manner as described earlier (where Z' is already chosen). 
Changes in disparity are linear with changes in camera separation so the current camera separation must be scaled 
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to match the disparity range in the captured images to the disparity range permitted on the display, After this the image 
cropping must be altered to adjust the ZDP so that the objects appear at the correct depths, A proportion of the image 
must be cropped from the left side of the left image and the right side of the right Image. 
[01 26] Cropping any amount from the input images will alter the proportion of the image represented by the disparity. 
5 In fact the actual disparity scales by; 

(7/(7-crop proportion)). 

10 [0127] After solving the equations governing this relationship the scale factor is: 

S=-a^(ci^(dt^^ D-d^) [34] 

'5 [0128] The proportion of the images to crop is calculated as: 

Cropping = -Sd^' [35] 

[0129] If A is known, either input by the user or calculated in using the methods described earlier, the scaled vaiue 
(S'A) may be output for the user. 

[0130] If the images used to obtain the disparities have already been cropped the simplest solution is to calculate 
the disparity proportions as a proportion of the uncropped image. Otherwise the camera separation, as a proportion 
of the image width, must be Itnown. This assumes knowledge of Z', W etc. If these are known the uncropped width is 
known and the method aiready described is used. The cropping specified by equation [32] is to be taken off the un- 
cropped images and may be more or less than the cropping applied to the images initially. 
[0131] An alternative way of matching Image disparity to display disparity, if the camera separation (A) cannot be 
changed (perhaps in a fixed head camera), is to alter the focal length of the camera (or field-of-view). 
[0132] The disparity of a point at a distance N is: 



so with A and N' fixed, scaling d,^ by a factor S requires scaling the focal length, f, by the same factor. 

[0133] The scale factor required to alter cf^ is calculated using equation [34], and equation [35] again gives the 

cropping required to adjust the disparity to fit the correct range on the display 

[0134] Physical limits of the camera will now be discussed, 

[0135] With real cameras there will be a limit on the range of focal lengths the lens may be set to and also on the 
camera separation. A real system using these protocols would have to recognise when the required settings are outside 
the limits of the hardware and indicate this to the user. 

[01 36] use of the method with moving cameras, a changing scene or video image will now be discussed. 
[01 37] If the scene is changing there are two choices. 

1 . Decide on the maximum depths which are likely to be seen and determine one set of camera parameters used 
throughout. 

2. Determine the depth visible in the scene before taking each new stereoscopic picture. 

[0138] Option 1 ) is limited in that it requires some decision about the range of depths which will be present in the 
scene. If the scene changes are unpredictable the depths may exceed this and the stereoscopic image will potentially 
be uncomfortable to view. 

[0139] For option 2) some method of determining the depth in a scene is required. For a computer graphics system 
using a depth buffer, as is common with most 3D graphics rendering methods, would allow reading of all the depth 
displayed in a frame. The depth buffer for one frame could be used to determine the depth likely to be present in the 
next frame. This is quite effective when the scene changes are gradual. 

[0140] Another alternative, with a computer graphics system, is to render the current frame once, before rendering 
the stereoscopic version. This pre-rendering could be used to provide the depth information. Most graphic effects, such 
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as texturing and lighting, wliich slow down rendering, could be turned off since the image is not intended for viewing. 
Also the image could be drawn smaller than the required image to further reduce the rendering time. 
[0141] With any depth measuring system there are lil<ely to be jumps in camera separation or zoom as objects enter 
and leave the measurement space. Using an expanded field of view with the pre-rendering method would allow pre- 
s diction of changes in depth likely to happen in later frames, allowing the changes in zoom or camera separation to be 
smoothed over time, reducing the stress caused to the viewer. 

[0142] Where the stereoscopic images are being captured by real cameras (possibly video cameras) some form of 
range finder might be used to measure the distances as each frame is shot, eg camera auto-focus system or laser 
range finder. Alternatively, the images may be processed using computer vision methods to measure the depths present 
10 from frame to frame, in order to estimate the parameters for the next frame. Again, using a range finder of some form 
will allow prediction of future depth changes. 

[0143] Processing stereoscopic images to determine the depth range presented in them gives the possibility of al- 
tering the images to present new views of the same scene, without taking more photographs. 
[0144] Use of focus information In a real camera will now be discussed. 
is [0145] The Kodak DCS 3 digital camera has a 'Depth of field' focusing method whereby the photographer focuses 
on the near object, the far object and then frames the scene and takes the photograph. The camera ensures that the 
shutter speed and aperture are set so that the depth of field in the image matches the near and far focus distances 
selected by the photographer. This relates only to measuring a depth range using a camera, not to setting depths for 
a stereoscopic display 

20 [0146] Information from any similar auto-focus system in a real camera may be used to capture some of the details 
required for the calculations. This can be helpful when the distances to near and far points in a scene may be difficult 
to measure directly, especially if the distances are large. 
[0147] One way of using this information would be to focus three times; 

25 1 . Focus on the nearest point and store that information (possibly by a half press of the shutter release). 

2. Focus on the furthest point and store that (another half click) 

3. Set up the desired shot, trying to avoid moving the camera too far. Adjusting FOV/f to the required value and 
make sure the camera is correctly oriented. 

30 [0148] The distances measured by the autofocus system would then be used as the input to the camera protocols 
as the distances to the nearest and furthest points. The lens focal length would be taken automatically from the lens, 
or adjusted automatically in the case of a camera where lens separation is set by the user and ^ adjusted to match, 
The only remaining details to be entered then are the display parameters. Either a set of defaults could be built in, 
perhaps with selection from a menu, or if the camera is programmable, new display settings could be downloaded to It, 

35 [0149] It may be possible to measure the photographer's eye separation by asking him/her to move the camera 
lenses, of a multi-head camera, so they (or markers on them) are in front of their eyes, and taking a measurement of 
the separation. 

[0150] There are many methods that could be used to measure the real depth in a scene, these include: 

Range finders of any type, auto focus mechanism on a real camera, tape measure, disparity estimation from stereo- 
•"o scopic image pairs, laser/ultrasonic rangefinder. 

[0151] With CG software, the depths could be found by ray-tracing, use of the depth buffer, computing distances 

directly to the objects, or to their bounding boxes. Many solutions are available to find nearest and furthest the surface 

in the computer graphics literature. 

[0152] Physical camera calibration will now be discussed. 
45 [0153] Care must be taken when converting entered or calculated focal length values to field-of-view values, that 

accurate parameters are used for the conversion. 

[0154] Any real camera used fortaking stereoscopic photographs using the methods described should be calibrated 
accurately for the true focal length. If the true focal length is not the same as the reading from the lens, incorrect 
separation or focal lengths may be calculated, producing errors in the resulting images. 
50 [0155] With digital cameras a width for the CCD is specified though in reality the full width may not be used. This 
must be taken into account when entering details into the camera separation methods. 

[0156] One method for calibrating a camera is to photograph a ruler (crossing the entire field of view horizontally) at 
a known distance from the camera. The width of the photographed portion (L) combined with the distance of the ruler 
from the camera (C) allow calculation of the field of view. The effective fiim/CCD width at any given focal length (read 
55 from the lens/imaging software) is also easily calculated: 

e = 2tan"^ (L/2C) 
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W^Lf/C 

[01 57] Note also that the calibration will change as the focal length of the camera lens Is changed, so e and W, should 
5 be calculated at several different zoom settings. 

[0158] Adjusting the Images when viewed if tlie display used for view/ing is different from the display assumed during 
calculation of the camera parameters, w\\\ now be disclosed. 

[0159] If the display used for viewing is different, or even the eye separation of the viewer is known to be different, 
the images may be adjusted after capture to adjust the perceived depth on the new display. This is lil<eiy to reduce the 
10 image quality, since the images may be scaled/cropped, so it would usually be best to capture the correct image in the 
first place. 

[01 60] To accomplish the adjustment the method described to adjust the camera separation based on disparities in 
a stereoscopic pair is used. The disparities in the images to be viewed must be provided, either by having stored them 
with the image pair, or analysing the images to find the disparities. 
'5 [0161] Equation [34] is then used to obtain a scale factor for the images. This scale factor is used to reduce (or 
increase) the size of the image on screen, reiative to the full screen size. Equation [35] can then be used to provide 
an offset for the images to ensure the perceived depth fits the required range for the new display. 
[0162] Multi-view image capture systems will now be discussed. 

[01 63] For muiti-view image capture systems the focal length of all cameras must be identical, otherwise some image 

20 pairs will have vertical disparity, potentially causing discomfort to the viewer. 

[0164] Also, Images in a multi-view system may be viewed by the left eye in one situation and by the right eye when 
the viewer has moved across the display. This means the Images cannot be cropped to give any desired perceived 
depth distribution - all the depth must appear in front of the display plane, It is possible to adjust the amount displayed 
by altering the camera separation according to the depth visible in each image pair. Any variations in camera separation 

25 due to scene depth changes must be gradual, (as for the case of dynamic camera control). 

[0165] If the dispiay is tracked it would be possible to arrange for the images to be cropped in real-time In order to 

adjust the perceived depth range. 

[0166] Avoiding 'frame-cancelling' will now be discussed. 

[0167] Frame-cancelling occurs when objects which should appear in front of the display plane are at the edge of 
30 the images. The brain interprets the depth of an object which goes off the edge of the images as being behind the 
screen border, and so behind the screen plane in depth because it should be visible if it was in front. This can produce 
an uncomfortable image even if the perceived depth range is within the defined limits for the display. 
[0168] This confilctcan be avoided by detecting such areas and adjusting the scene Input parameters to put objects 
at the screen edge behind the display plane. The image should be analysed in order to determine the distance to the 
35 closest object at the image edge and to set 2' to this distance. Alternatively any other measurement method could bo 
used either within, or external to, the camera. 

[0169] If disparity at the screen edge is measured, or the images are to be adjusted after they have been taken, the 
method described earlier for adjusting camera parameters when disparity is defined would be used, d^' is the smallest 
disparity of objects at the edge of the images and dp' is also measured from the image, djg would be set to 0 and dp 
40 calculated, as before, from the display. 

[0170] Using approximations when accurate measurement is difficult wiil now be discussed. 

[0171] In situations where it is difficult to make accurate measurements approximations may be used to give useful 

results 

45 * A 'typical' viewer is usually taken to have an eye separation of 60mm 

• Unless tracked the actual distance of the viewer from the display (Z) is unknown, the design viewing distance 
would be used, 

• Precalculated sets of camera parameters for broadly applicable situations may be used - e.g. compute cameras for 

50 • 'distant objects' (perhaps scene depth from 1 000m to infinity) 

• 'far objects' (perhaps scene depth from 100m to infinity) 

• 'mid range' (perhaps scene depth from 1 0m to 1 00m) 

• 'close range' (perhaps scene depth from 0,5m to 10m) 

55 [0172] The depth mapping will not be the exact mapping expected if measurements are accurate but it is simple to 
estimate the appropriate depth range to ensure all depth actually present in the scene is within the limits of the broader 
categories listed. 
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Claims 

1 . A method of producing a stereo image of a (real or simulated) scene using at least one (real or simulated) camera, 
whicti creates ttie impression of being a 3D image when viewed on a display by a user, wherein in each position 

5 at which an image is captured the camera axis is parallel to the camera axis at all other positions, and the depth 
of the scene is mapped onto a maximum perceived depth of the image on the display, and the maximum perceived 
depth is chosen to provide comfortable viewing for the user. 

2. A method as claimed in claim 1 , wherein the method makes use of parameters of the display, including the maximum 
10 perceived depth of an object in front of the display N, and the maximum perceived depth of an object behind the 

display F, and also makes use of the user's eye separation E, and the distance Z of the viewer from the display. 

3. A method as claimed in claim 2, wherein the distance Z' from the camera to the Zero Disparity Plane is calculated 
based on the values of N and F, and also on the values of the distance N' from the camera to the closest surface 

is in the scene, and the distance F' from the camera to the furthest surface in the scene. 

4. A method as claimed in claim 2, wherein the distance Z' from the camera to the Zero Disparity Plane is specified, 
and the values of N' and F' (as defined in claim 3) are also specified, and wherein the most suitable of N' or F' is 
kept fixed and a new value is calculated for the other of N' or F' based on the values of N and F 

20 

5. A method as claimed in claim 2 or 3, wherein any one of N', F' and Z' (as defined In claim 3) is calculated based 
on the values of the other two. 

6. A method as claimed in any one of claims 2 to 5, wherein the maximum crossed and uncrossed disparities dN and 
25 dF, are calcLilated from N, F, the separation E of the user's eyes, and the distance Z from the user to the display. 

7. A method as claimed in any of claims 2 to 6, wherein the camera separation A used to produce the stereo image 
is calculated based on N and F to achieve the desired perceived depth (N + F). 

30 8. A method as claimed in claim 7, wherein the field of view of the camera is adjusted to allow for that part of the 
image which will be cropped as a result of the use of parallel camera positions. 

9. A method as claimed in any one of claims 2 to 8, wherein the camera separation is fixed, and the desired focal 
length of the camera is calculated to achieve the desired perceived depth (N + F). 

35 

10. A method as claimed in any preceding claim, wherein the focal length f of the camera is taken into account when 
measuring the values of N' and F' (as defined in claim 4). 

1 1 . A method as claimed in any preceding claim, wherein the camera separation A is limited to the user's eye separation 
"0 E multiplied by the scale factor S as herein defined. 

12. A method as claimed in claim 1 , wherein a stereoscopic pair of images is created using a given camera separation, 
the images are analysed to find the disparities between homologous points in the images, and said disparities are 
then used to adjust said camera separation. 

45 

13. A method as claimed in Claim 12, wherein said stereoscopic pair of images is created for display on a first display, 
and wherein said camera separation is adjusted to make the images suitable for display on a second display. 

14. A computing device adapted to carry out a method as claimed in any preceding claim. 

50 

15. A camera or camera system comprising a computing device as claimed in claim 14. 
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Perceived depth on a stereoscopic 3D display systems 
FIGl 



Virtual display in world space or 
zero disparity plane (ZDP). 
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Cropping for 
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Parallel cameras and imaging adjustment requirements 
FIG 2 
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Display 




Viewing geometry of a stereoscopic display 
FIGS 




Virtual display 



Mapping the display to the scene being captured 
FIG 4 
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Enter display details 
including W,N,F,Z,E. 



Enter scene details 
including N',F,Wf,f 
adjust NTand F'by subtracting f. 



X 



For the given parameters,calculate the maximum crossed and 
uncrossed disparities allowed. 
dN=NE/(Z-N) 
dF=FE/(Z+F) 



Calculate the ZDP distance 
R=dN/dF 
71= (R+l)/(l/N'+R/F') 



Calculate the virtual display \vidth from the 
ZDP and the field of view 
(possibly obtained from the focal length) 
(W'+A) = Z'Wf /f 



Calculate scale factor between display 
and world-ratio of calculated widih to 
display width plus camera separation 
scaled back to display space 
S=Z'Wf /f (W+dNNV(Z'-N')) 



X 



Calculate camera separation from 
scale factor,max crossed disparity, 
near distance and ZDP distance. 
A=SdNN/(Z'-N') 



Once converged,calculate portion 
of image to crop 
=A/(W'+A) 



Output 
A,crop 



Flow diagram,calculating camera separation where focal length is fixed 
FIG 5 
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Enter display details 
including W,N,F,Z,E. 



Enter scene details 
including N',F,Wf,f 
adjust N'and Pby subtracting f. 



For the given parameters,calculate the maximum crossed and 
uncrossed disparities allowed. 
dN=NE/(Z-N) 
dp=FE/(Z+F) 



X 



Calculate the ZDP distance 
R=dN/dp 
Z'= (R+l)/(l/N'+R/F') 



X 



Calculate the virtual display width from the 
ZDP and the field of view 
(possibly obtained from the focal length) 
W = Z'Wf /f 



Calc scale factor between display 
and world-ratio of calculated width to display 
S=Z'Wf/fW 



Calculate camera separation. 
A^Sd^NYCZ'-N') 



Calculate new focal length 
G=2tan-' ((W'+A)/2Z') 
f=Wf/2tan(0/2) 



Correct Separation by using the new f 
to adjust N'and F'in calculation of Z' 
Re-compute the separation 



FIG 6 



Once converged,calculate portion 
of the image to crop 
= A/(W+A) 



Output A, 
e,f,crop 

Flow diagram,calculating camera separation where focal length is not fixed 
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Enter display details 
including W,NJ^,Z,E. 



Enter scene details 
including N',F,Wf,f 
adjust N'and Fby subtracting f. 



For the given parameters.calculate the maximum crossed and 
uncrossed disparities allowed. 
dN-NE/(Z-N) 
dF=FE/(Z+F) 



Calculate the ZDP distance. 
Z'= (R+l)/(l/N'+R/F') 



T 



Calc scale factor between display and world- 
ratio of world disparity to display disparity. 
S=d '/d^. 



T 



Calculate the virtual display width from 
the scale factor. 
W'=SW 



Calculate new focal length 
0=2tan -'((W'+A)/2Z') 
^Wf/2tan(0/2) 



Correct 0 arad fby using the new f 
to adjust N'and F'in calculation of Z' 
Re-compute the separation. 



X 



Once converged,calculate portion 
of image to crop 
=A/(W'+A) 



jDutput 0, 
f,crop 



Flow diagram,calculating focal length where camera separation is fixed 
FIG 7 
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